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ABSTRACT 

This  report  summarises  key  outcomes  from  a  half-day  workshop  on  Simulation  in  Training 
held  as  part  of  the  Defence  Human  Sciences  Symposium  (DHSS)  in  November  2012.  The 
workshop  involved  presentations  on  simulation-based  training  (SBT)  research  within  DSTO, 
and  syndicate  group  discussions  aimed  at  addressing  key  questions  relating  to  SBT.  The 
workshop  outcomes  highlighted  the  need  for  the  research  community  to  obtain  more 
evidence  regarding  the  cost-effectiveness  of  simulation  and  for  the  defence  client  to  adopt 
more  rigorous  processes  when  acquiring  and  evaluating  simulation.  At  present,  knowledge 
gaps  are  limiting  the  research  community's  ability  to  provide  Australian  Defence  Force  (ADF) 
customers  with  sound  advice  regarding  the  effectiveness  of  SBT  and  for  the  ADF  to  fully 
realise  the  benefits  associated  with  SBT. 
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Executive  Summary 

Defence  forces  worldwide  are  seeking  to  increase  the  use  of  simulation  for  training  and 
education.  However,  stakeholders  within  the  defence  and  research  communities,  as 
well  as  industry,  are  asking  questions  which  indicate  a  level  of  uncertainty  regarding 
the  best  use  of  simulation.  These  questions  relate  to  where  in  the  training  continuum 
simulation-based  training  (SBT)  should  be  used,  the  optimum  mix  of  live  training  and 
SBT,  and  the  required  level  of  simulator  fidelity.  However,  opportunities  for 
collaboration  between  stakeholders,  which  might  help  to  address  these  questions,  are 
limited.  One  such  opportunity  arose  during  the  2012  Defence  Human  Sciences 
Symposium  (DHSS)1.  Accordingly,  a  workshop  entitled  "Simulation  in  Training"  was 
conducted  during  DHSS  2012  with  the  aim  of  discussing  key  issues  for  SBT.  The 
workshop  was  attended  by  27  personnel  from  DSTO  and  other  defence  agencies,  along 
with  representatives  from  academia. 

The  workshop  involved  a  series  of  presentations  on  SBT  research  within  DSTO, 
followed  by  syndicate  group  discussions  focussed  on  (1)  providing  advice  to  the  client 
on  the  use/  acquisition  of  simulation,  (2)  engaging  with  industry  regarding  simulation 
development,  (3)  cost-effectiveness  of  SBT,  and  (4)  future  research  /  high  pay-off  areas 
for  SBT.  The  workshop  outcomes  included  an  increased  awareness  of  SBT  research 
within  DSTO,  the  identification  of  key  issues,  challenges  and  lessons  for  SBT  research, 
as  well  as  providing  an  increased  network  of  contacts  involved  in  SBT  research.  The 
workshop  also  highlighted  (1)  deficiencies  in  the  acquisition  and  use  of  SBT  within  the 
Australian  Defence  Force  (ADF)  and  (2)  limitations  in  the  advice  researchers  can 
currently  provide  to  defence  clients  regarding  the  best  use  of  simulation.  The 
implications  of  this  are  that  the  ADF  may  not  be  maximising  the  potential  benefits  of 
SBT,  and  that  researchers  and  clients  need  to  identify  the  specific  studies  and  resources 
required  to  start  addressing  this  situation.  Based  on  the  workshop  outcomes,  a  number 
of  lessons  were  identified  for  both  the  research  community  and  the  defence  client: 

•  More  research  is  needed  to  inform  our  understanding  of  the  cost-effectiveness  of 
SBT,  including  a  better  understanding  of  the  relationship  between  simulator 
fidelity  and  training  effectiveness,  determining  which  skills  are  best  trained  using 
SBT,  and  developing  more  reliable  measures  of  effectiveness. 


1  The  Defence  Human  Sciences  Symposium  is  the  annual  meeting  of  the  DSTO  Human  Sciences  Hub 
which  comprises  researchers  from  DSTO,  as  well  as  academics  and  practitioners  from  the  wider  scientific 
community  with  an  interest  in  human  sciences  research. 
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Principles  of  training  and  education  such  as  guided  instruction  and  consideration 
of  individual  differences  (e.g.,  ability,  learning  styles)  are  important  when 
delivering  SBT. 

There  are  likely  to  be  benefits  from  using  SBT  to  train  for  high-risk  missions,  and 
the  development  of  critical  thinking,  decision  making  skills,  and  other  non-kinetic 
skills  (e.g.,  cultural  awareness). 

There  is  a  need  for  defence  clients  to  follow  more  rigorous  processes  when 
acquiring  a  simulation  capability,  including  better  definition  of  training 
requirements  and  simulation  specifications  and  using  cost-benefit  analysis  to 
compare  training  options. 
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1.  Introduction 


On  13  November  2012,  DSTO  Land  Operations  Division  (LOD)1  hosted  a  half-day 
workshop  on  'Simulation  in  Training'  as  part  of  the  2012  Defence  Human  Sciences 
Symposium2.  The  main  driver  for  the  workshop  was  to  address  an  enduring  request  from 
defence  clients  for  advice  on  how  to  best  use  simulation  within  the  training  continuum.  At 
present  defence  forces  are  seeking  ways  to  increase  their  use  of  simulation  in  order  to 
deliver  more  effective  and  efficient  training.  While  several  groups  within  the  Australian 
Defence  Organisation  and  the  wider  academic  community  are  attempting  to  tackle  these 
issues,  much  of  the  work  is  being  done  in  isolation.  The  workshop  provided  an 
opportunity  to  discuss  these  issues  in  person  with  key  stakeholders  who  were  attending 
the  symposium. 

Within  Australia,  key  stakeholders  in  the  simulation-based  training  (SBT)  domain  include 
various  groups  within  the  Defence  Science  and  Technology  Organisation  (DSTO),  the 
Australian  Defence  Force  (ADF),  Australian  Defence  Simulation  Office  (ADSO),  the  wider 
scientific  community,  and  Defence  Industry.  The  objective  of  the  workshop  was  to  bring 
together  personnel  from  these  groups  to  identify  key  issues,  challenges  and  lessons  from 
the  use  of  simulation  in  training,  in  order  to  establish  a  basis  for  further  discussion  in  the 
area,  and  ultimately,  to  provide  informed  advice  to  clients  regarding  the  use  of  SBT. 

This  report  summarises  the  key  outcomes  from  the  workshop  based  on  notes  collated  from 
individual  presentations  and  syndicate  group  discussions.  The  report  is  aimed  primarily  at 
the  research  community  to  inform  future  research  in  support  of  defence  clients;  however 
implications  for  defence  clients  and  industry  are  also  highlighted. 


2.  Workshop  Outline 

The  workshop  was  attended  by  27  personnel  who  were  mostly  from  DSTO  and  other 
defence  agencies,  with  some  representation  from  academia  (See  Appendix  A  for  the  list  of 
attendees  and  their  affiliation).  The  majority  of  attendees  were  either  working  in,  or  had  an 
interest  in,  SBT  research  and  most  were  responsible  for  providing  advice  to  decision¬ 
makers  regarding  the  acquisition  and  use  of  simulation  for  training.  The  workshop 
program  consisted  of  an  opening  address,  several  individual  presentations,  and  syndicate 
group  discussions.  Each  of  these  activities  is  described  briefly  below. 


1  The  workshop  was  facilitated  by  Dr  Ashley  Stephens  with  assistance  from  Mr  Philip  Temby 
(LOD)  and  Dr  Susannah  Whitney  (LOD). 

2  The  DHSS  is  the  annual  meeting  of  the  DSTO  Human  Sciences  Hub  which  comprises  researchers 
from  DSTO  and  academics  and  practitioners  from  the  wider  scientific  community.  The  symposium 
and  workshop  were  held  at  the  Mawson  Centre  in  Adelaide,  South  Australia. 
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2.1  Opening  Remarks 

The  workshop  commenced  with  the  facilitator  giving  a  short  presentation  that  outlined  the 
context  for  the  workshop,  as  well  as  the  scope  of  SBT  within  the  ADF.  It  was  noted  that  the 
scope  includes:  training  and  education,  live,  virtual  and  constructive  (LVC)  simulation, 
individual  and  collective  training,  and  distributed  versus  co-located  training.  In  addition, 
the  facilitator  highlighted  that  the  following  questions  are  typically  being  asked  by  defence 
clients  in  relation  to  SBT. 

1.  What  are  the  advantages  and  disadvantages  of  SBT? 

2.  How  do  we  measure  the  effectiveness  of  SBT? 

3.  Where  in  the  training  continuum  should  SBT  be  used? 

4.  What  is  the  optimum  mix  of  live  and  simulation-based  training? 

5.  What  level  of  (simulation)  fidelity  is  required? 

Due  to  the  limited  time  available,  it  was  not  expected  that  workshop  attendees  would  be 
able  to  answer  these  questions.  However,  they  were  highlighted  to  raise  awareness  of  the 
questions  being  asked  by  defence  clients,  and  to  encourage  attendees  to  think  about  how 
they  might  answer  them.  It  was  also  envisaged  that  the  questions  would  provide  a  starting 
point  for  further  discussion  within  the  research  and  defence  communities. 

2.2  Individual  Presentations 

Six  presentations  on  SBT  were  given  by  DSTO  staff  in  order  to  share  lessons  from  DSTO 
research  with  the  workshop  participants.  The  presentations  covered  a  range  of  areas  (e.g., 
individual  and  collective  training,  military  education),  different  simulations  (e.g.,  first- 
person  shooter,  microworlds,  flight  simulators)  and  different  domains  (e.g.,  air,  land).  The 
list  of  presenters  and  their  presentation  titles  are  shown  in  Table  1. 


Table  1:  List  ofzvorkshop  presenters  and  presentation  titles 


Presenter  and  Affiliation 

Presentation  Title 

Dr  Ashley  Stephens  (LOD) 

Simulation  in  Training 

Dr  Susannah  Whitney  (LOD) 

Use  of  Simulation  in  Individual  versus  Collective 

Training 

Dr  Greg  McLean  (AOD) 

Exercise  Black  Skies  12:  Evaluation  of  the  effectiveness  of 
synthetic  collective  training  for  an  Airborne  Early 

Warning  &  Control  team 

Dr  Greg  McLean  (AOD) 

The  Integration  of  Simulation  within  an  ab  initio  Pilot 
Training  Program 

Mr  John  Hansen  (C3ID) 

Professional  Military  Education 

Dr  Daniel  Goodburn  (C3ID) 

The  State  of  Simulation  within  Education 
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2.3  Syndicate  Group  Discussions 

Following  the  individual  presentations,  workshop  participants  were  split  into  three 
syndicate  groups.  The  purpose  of  this  session  was  for  participants  to  discuss  and  generate 
responses  to  the  following  questions. 

1.  What  key  messages  would  you  provide  to  clients  regarding  the  use  /  acquisition  of 
simulation  for  training? 

2.  When  is  simulation-based  training  cost-effective:  what  does  effective  mean  to  you? 

3.  Flow  would  you  engage  with  industry  in  order  to  achieve  best  return  on  investment 
for  simulation-based  training? 

4.  Where  should  future  research  in  simulation-based  training  focus?  Where  are  the  high 
pay-off  areas,  and  why? 

These  questions  were  selected  by  the  workshop  organisers  for  the  following  reasons:  (1) 
they  were  sufficiently  high  level  to  enable/ encourage  all  workshop  participants  to 
contribute  to  the  discussions3;  (2)  they  were  consistent  with  the  questions  being  asked  by 
defence  clients  regarding  SBT;  (3)  they  focused  on  identifying  key  messages  for  clients  and 
industry;  and  (4)  they  focused  on  addressing  future  areas  for  SBT  research. 

Participants  were  given  40  minutes  to  discuss  the  questions  in  their  groups  and  were  then 
asked  to  report  back  to  the  main  group  their  discussion  outcomes  (20  minutes).  Due  to  the 
limited  time  available,  it  was  not  expected  that  the  questions  could  be  addressed  in  any 
great  depth,  but  rather  provide  an  opportunity  to  collate  key  points  for  further  discussion 
and  follow-up  with  those  interested  in  SBT  after  the  workshop. 


3.  Key  Outcomes 

3.1  Individual  Presentations 

The  following  sections  summarise  key  lessons  highlighted  in  the  presentations  of  recent 
DSTO  research  examining  the  use  of  SBT.  More  details  about  each  of  the  DSTO  studies  are 
provided  in  Appendix  B. 

•  More  empirical  research  is  needed.  It  was  highlighted  by  one  presenter  that  one  of  the 
difficulties  in  providing  reliable  advice  to  the  client  results  from  the  broad  range  of 
areas  covered  by  SBT.  The  implication  is  that  the  effectiveness  of  SBT  may  need  to  be 
evaluated  for  each  specific  case  of  interest;  lessons  generated  from  one  case  may  not 
generalise  to  other  cases.  Consequently,  a  large  body  of  knowledge  is  required  in  order 
to  confidently  provide  meaningful  advice  to  the  client  regarding  the  use  of  SBT.  At 
present,  there  remains  a  lack  of  empirical  research  into  SBT  to  inform  such  advice. 


3  This  was  an  important  consideration  given  that  not  all  workshop  attendees  were  undertaking 
research  in  SBT  but  had  an  interest  in  the  area. 
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•  SBT  appears  to  be  more  effective  for  certain  skills.  It  was  highlighted  by  another 
presenter  that  the  utility/  effectiveness  of  SBT  can  be  dependent  on  the  skill(s)  being 
trained.  In  one  DSTO  study,  the  VBS2  simulation  was  found  to  be  ineffective  for 
collective  training  in  a  specific  collective  context  (i.e.,  section  attack  training  for 
infantry  teams)  but  showed  promise  for  individual  training  in  a  different  collective 
context  (e.g.,  hazard  awareness  training  for  individual  commanders  of  a  vehicle 
convoy).  The  implication  is  that  it  is  important  to  specify  the  task  and  context  in  which 
the  training  is  conducted  and  not  simply  describe  a  simulation  as  being  effective  (or 
not  effective)  for  training.  It  also  implies  that  in  some  cases,  SBT  may  be  more  effective 
when  training  individuals  in  a  collective  context,  than  training  teams  in  a  collective 
context.  Such  principles  may  be  helpful  for  defence  clients  to  consider  when 
employing  SBT;  however,  more  case  studies  are  needed  before  definitive  conclusions 
can  be  made. 

•  Limitations  in  simulator  fidelity  can  influence  SBT  effectiveness.  The  ability  to 
replicate  aspects  of  the  real-world  in  a  simulation  can  be  critical  to  SBT  effectiveness. 
For  example,  limited  field-of- views  on  desktop  displays  can  reduce  an  individual's 
level  of  situational  awareness  during  training,  especially  when  they  need  to  know 
where  their  other  team  members  are  located  in  the  virtual  environment.  The 
implication  here  is  that  simulator  fidelity  needs  to  support  the  tasks  being  trained  to  a 
sufficient  level;  otherwise  it  will  limit  the  degree  of  training  effectiveness  that  is 
achieved.  It  also  implies  that  where  the  requirement  for  situation  awareness  is  high 
(and  cannot  be  supported  by  simulation)  then  other  training  methods  may  be  more 
appropriate. 

•  SBT  may  not  always  be  cheaper.  There  is  often  a  perception  that  SBT  is  cheaper  than 
live  (or  traditional)  training  methods;  this  is  particularly  true  in  the  air  domain. 
However,  in  one  DSTO  study  it  was  found  that  there  was  little  difference  in  cost 
between  an  all  live  and  a  blended  training  program  involving  simulation  for  pilot 
training.  The  lesson  here  is  that  it  should  not  be  assumed  that  the  use  of  SBT 
automatically  achieves  cost-savings.  Furthermore,  there  can  be  hidden  costs  associated 
with  delivering  SBT,  such  as  costs  associated  with  hardware  and  infrastructure, 
software  development  and  specialist  support  services;  some  of  which  may  be 
significant  and  ongoing  (e.g.,  maintenance  and  development  costs). 

•  Realistic  SBT  can  be  resource  intensive.  It  was  noted  by  one  presenter  that  SBT  can  be 
effective  for  improving  collective  skills  and  mission  performance  in  the  air  domain. 
However,  to  achieve  such  outcomes,  a  4:1  ratio  of  white  force  to  exercise  participants 
was  required.  The  implication  is  that  conducting  realistic  SBT  can  be  highly  resource 
intensive,  and  may  not  be  achievable  on  a  frequent  basis  if  the  client  incorporated  SBT 
into  their  regular  training  program.  Further  research  is  required  to  examine  whether 
this  ratio  can  be  reduced  (e.g.,  through  greater  use  of  simulated  entities,  using  non¬ 
expert  role  players)  without  compromising  training  effectiveness. 
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•  Opportunities  for  using  simulation  are  being  missed.  Another  presenter  highlighted 
that  a  recent  review  into  the  state  of  simulation  in  the  field  of  education4  indicated  that 
(1)  developments  in  simulation  tends  to  focus  on  training  rather  than  education,  (2) 
there  is  a  lack  of  focus  on  using  simulation  to  help  trainees  develop  critical  thinking 
skills,  creativity  and  the  ability  to  deal  with  ambiguity,  and  (3)  there  is  a  gap  between 
research  in  instructional  design  for  simulation-based  education  and  simulation 
development.  The  implication  of  these  issues  is  that  there  are  opportunities  being 
missed  in  developing  and  using  simulation  to  support  education  of  military  personnel. 
It  was  highlighted  that  microworld  simulations  may  be  useful  tools  for  addressing  this 
issue,  by  raising  awareness  of  different  thinking  styles  and  for  training  skills  such  as 
adaptability.  There  is  currently  some  research  within  DSTO  and  overseas  defence 
agencies  that  is  examining  the  utility  of  such  tools  for  training  and  education  purposes. 

3.2  Syndicate  Group  Discussions 

Following  the  presentations,  the  workshop  attendees  were  split  into  three  syndicate 
groups.  The  key  points  and  themes  raised  by  the  groups  to  each  of  the  four  questions  are 
summarised  in  the  following  sections. 

What  key  messages  would  you  provide  to  clients  regarding  the  use  /  acquisition  of 
simulation  for  training? 

Participants  highlighted  several  points  in  relation  to  this  question.  Overall,  participants 
acknowledged  that  SBT  has  potential  advantages  and  there  is  much  anecdotal  evidence 
regarding  the  utility  of  SBT.  However,  participants  noted  that  there  remains  a  lack  of 
empirical  research  into  the  effectiveness  of  SBT,  which  was  perceived  to  be  a  major 
impediment  in  providing  meaningful  advice  to  clients.  In  particular,  it  was  noted  that 
there  appears  to  be  a  lack  of  case  studies  demonstrating  clear  evidence  for  the  benefits  of 
SBT.  The  implication  of  this  situation  is  that  clients  may  not  be  using  simulation  in  the  best 
possible  way;  in  the  worst  case,  it  is  possible  that  SBT  may  lead  to  negative  training.  It  also 
implies  that  there  is  a  need  for  researchers  and  clients  to  work  together  to  identify  the 
specific  studies  and  resources  required  to  start  addressing  this  situation. 

In  terms  of  simulation  acquisition,  participants  made  some  specific  points  regarding  the 
need  for  the  client  to  take  a  more  rigorous  'systems'  approach  to  the  acquisition  of  SBT. 
This  means  that  the  contribution  SBT  makes  to  overall  capability  needs  to  be  understood 
(as  with  all  training);  in  this  regard,  the  acquisition  of  simulation  should  potentially  be 
treated  like  any  other  capability.  Consequently,  participants  believed  that  the  acquisition 
process  needs  to  include  the  rigorous  specification  of  simulation  requirements  and  be 
based  on  a  Training  Needs  Analysis  (TNA).  However,  some  participants  questioned  the 
quality  of  TNAs  being  conducted  within  Defence,  and  noted  that,  in  some  cases,  there 
appeared  to  be  no  TNA  conducted  at  all.  This  is  clearly  an  impediment  for  Defence  in 
terms  of  getting  value-for-money  from  SBT;  if  training  requirements  are  not  being 


4  Magee,  M.  (2006).  A  State  of  the  Field  Review:  Simulation  in  Education  Final  Report.  Alberta 
Online  Learning  Consortium.  Calgary  Alberta,  Canada. 
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properly  articulated  as  part  of  a  TNA,  it  seems  unlikely  that  simulation  specifications  are 
being  properly  addressed. 

It  was  noted  that,  within  the  ADF,  the  acquisition  of  simulation  occurs  either  as  a  very 
minor  part  of  a  major  capability  acquisition,  or  as  a  minor  capability  acquisition  and  hence 
may  not  be  subject  to  the  same  level  of  rigour  as  a  major  capability.  Flowever,  participants 
felt  that  this  policy  is  based  on  an  argument  that  a  low  up-front  cost  equates  to  low-risk, 
which  may  not  be  justified,  particularly  when  taking  into  account  costs  throughout  the 
capability  lifecycle  (e.g.,  costs  associated  with  simulator  upgrades).  For  the  client,  there 
can  be  a  tension  between  following  a  formal  acquisition  process  for  simulation  acquisition 
(including  requirements  definition  and  TNA)  and  choosing  to  acquire  COTS  products. 
The  first  option  may  be  more  expensive  in  the  short  term,  but  is  more  likely  to  deliver 
cost-effective  training  in  the  longer  term.  Buying  COTS  products  is  more  convenient,  and 
possibly  quicker  and  cheaper  in  the  shorter  term,  but  may  result  in  training  shortfalls  and 
other  costs  in  the  longer  term. 

Participants  also  highlighted  that  Tow-technology'  approaches  can  provide  effective 
training  and  that  SBT  should  not  automatically  be  assumed  to  be  the  best  (or  cheapest) 
option.  Two  examples  were  provided  to  illustrate  this  point.  One  example  was  the  use  of 
small  group  discussions  to  'wargame'  contingency  plans  for  specific  scenarios  with 
aircrew  trainees.  It  was  highlighted  that  this  type  of  training  can  be  very  effective  in 
teaching  aircrew  tactics  and  teamwork.  The  second  example  was  the  use  of  mental 
checklists  (e.g.,  if  A  occurs,  then  do  B)  to  help  aircrew  trainees  cope  with  high  levels  of 
cognitive  workload  in  stressful  situations.  There  are  likely  to  be  other  examples  within  the 
ADF. 

Participants  also  highlighted  a  number  of  important  principles  and  concepts  that  could  be 
applied  when  using  SBT.  The  following  examples  were  provided: 

♦  Guided  instruction:  There  was  consensus  that  using  guided  instruction  is  an 
important  ingredient  for  effective  training.  It  was  acknowledged  that  using  simulation 
and  other  training  technologies  can  enable  more  efficient  training  and  increased 
training  throughput  (e.g.,  through  self-paced  learning).  Flowever,  participants  felt  that 
there  is  a  risk  in  relying  on  technologies  to  deliver  training  and  expecting  trainees  to 
automatically  assimilate  the  necessary  information;  in  some  cases,  trainees  will  require 
guided  instruction  and  feedback.  Traditionally,  this  is  provided  by 
instructors/  teachers;  however,  developments  in  areas  such  as  automated  intelligent 
tutoring  may  be  able  to  provide  this  capability  in  the  future. 

•  Training  beyond  competency  levels:  It  was  noted  that  due  to  resource  and  time 
constraints,  the  current  training  model  within  Defence  is  based  on  competency  based 
training.  This  approach  does  not  allow  the  degree  of  practice  required  to  achieve 
higher  levels  of  proficiency  (i.e.,  training  to  mastery),  nor  does  it  allow  opportunities  to 
assess  the  capability  limits  of  operators  during  training.  This  was  seen  as  a  shortfall  in 
current  training  practices  which  could  potentially  be  addressed  (in  part)  by  using  SBT. 
This  is  because  simulation  can  allow  additional  practice  without  incurring  significant 
additional  cost,  as  well  as  providing  a  safe  environment  in  which  to  push  operators  to 
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the  limits  of  their  performance  (e.g.,  driver  fatigue).  Consequently,  this  was  also 
identified  as  an  area  for  future  research  into  SBT. 

•  Training  versus  Education:  As  noted  in  Section  3.1,  there  appears  to  be  a  tendency  for 
simulation  development  to  focus  on  training  rather  than  education;  consequently, 
there  is  a  risk  that  SBT  may  focus  on  teaching  trainees  'what  to  think'  (e.g.,  through 
exposure  to  relevant  scenarios  and  having  to  respond  with  the  'correct'  response).  It 
was  acknowledged  that  training  personnel  'what  to  think'  may  be  appropriate  for 
certain  military  populations  and  under  certain  circumstances  (i.e.,  responding 
instinctively  to  certain  situations,  obeying  orders).  However,  in  some  cases,  personnel 
may  be  required  to  make  decisions  under  conditions  of  uncertainty  and  ambiguity, 
where  such  an  approach  is  not  effective.  Consequently,  there  is  a  need  to  educate 
personnel  in  'how  to  think'  (e.g.,  creative  thinking  and  problem-solving,  working 
through  different  courses  of  action  and  seeing  the  different  consequences.)  This  was 
also  identified  as  an  area  for  future  research. 

•  Realistic  training:  Current  military  training  models  advocate  a  'crawl-walk-run' 
approach  to  training;  that  is,  starting  with  simple  tasks  and  progressing  to  more 
difficult  tasks.  However,  the  effectiveness  of  training  practices  that  involve  getting 
learners  to  attempt  difficult  and  challenging  tasks  in  the  early  stages  of  training  (i.e., 
'throwing  them  in  the  deep  end')  was  discussed.  As  a  rule  this  practice  was  not 
recommended  by  participants  but  there  was  agreement  that  providing  trainees  with 
challenging  experiences  in  a  non-judgemental  learning  environment  (i.e.,  devoid  of 
formal  assessment  and  no  criticism  for  making  errors)  may  produce  good  learning 
outcomes.  This  concept  was  also  identified  as  an  area  for  future  research  into  SBT. 

When  is  simulation-based  training  cost-effective? 

Overall,  the  consensus  among  participants  was  that  there  is  currently  insufficient 
empirical  evidence  to  provide  a  definitive  answer  to  this  question.  Notwithstanding  this,  a 
number  of  issues  relating  to  the  evaluation  of  cost-effectiveness  were  highlighted. 

Participants  believed  that  the  critical  measure  of  training  effectiveness  was  real  world  ('on- 
the-job')  performance,  although  there  was  little  discussion  as  to  how  this  could  be  reliably 
measured.  It  was  felt  that  there  was  a  lack  of  good  measures  of  effectiveness  for  SBT;  one 
suggestion  was  to  apply  a  checklist  of  criteria  (including  cost,  risk  and  training  benefit)  to 
help  make  decisions  about  when  to  use  SBT.  It  is  worth  noting  that  while  this  suggestion 
seems  reasonable,  actually  determining  these  criteria  would  not  be  a  trivial  exercise. 

In  this  respect,  it  was  believed  that  simulation  should  be  considered  as  just  one  tool  for 
supporting  training  and  that  statements  regarding  cost-effectiveness  need  to  be  made 
relative  to  other  training  methods  (e.g.,  traditional  instruction,  live  training).  However, 
participants  indicated  that  there  was  a  lack  of  empirical  data  on  the  effectiveness  of 
conventional  training  methods,  which  limits  the  ability  to  make  reliable  comparisons  with 
new  training  methods  such  as  SBT.  This  is  an  important  point;  if  the  cost-effectiveness  of 
current  training  methods  is  not  fully  understood,  it  makes  it  difficult  to  make  informed 
decisions  regarding  new  training  methods,  including  the  acquisition  of  simulation. 
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In  addition,  it  was  noted  that  within  the  military  domain  there  was  a  strong  reliance  on 
subject  matter  expert  judgement,  with  a  lack  of  more  objective  measures.  This  was  thought 
to  be  an  impediment  to  the  progress  of  training  research,  as  well  as  limiting  the  ability  to 
evaluate  cost-effectiveness. 

Participants  also  raised  a  number  of  specific  questions  that  highlighted  gaps  in  our  current 
knowledge  of  SBT. 

•  How  do  skill  acquisition  and  retention  rates  for  SBT  compare  with  those  for  traditional 
training?  It  is  conceivable  that  skill  retention  is  different  for  SBT  compared  with 
conventional  methods;  however,  the  rate  of  skill  degradation  for  both  training 
methods  is  not  known.  Hence  at  present,  it  is  not  possible  to  compare  the  long  term 
effectiveness  of  SBT  and  conventional  training.  It  was  noted  that  the  long  term 
effectiveness  of  SBT  has  implications  for  the  scheduling  of  refresher  training  and  the 
overall  cost-effectiveness  of  SBT  relative  to  conventional  training. 

•  What  type  of  training  is  most  effective  for  a  specific  task?  And  what  level  of  simulation 
fidelity  is  required?  Participants  agreed  that  certain  skills  appear  to  be  more  suited  to 
SBT  than  others,  but  it  was  noted  that  the  relationship  between  simulation  fidelity  and 
training  effectiveness  is  not  fully  understood,  and  may  need  to  be  established  in  each 
specific  case.  This  is  an  important  issue,  as  higher  fidelity  usually  equates  to  higher 
cost;  however,  higher  fidelity  may  not  be  justified  unless  it  results  in  more  effective 
training. 

•  What  impact  do  individual  differences  (e.g.,  ability,  personality,  learning  styles)  have 
on  training  outcomes?  Does  SBT  benefit  everyone  equally?  Participants  were  not  able 
to  answer  these  questions  specifically  for  SBT.  However,  it  was  noted  that  a  'one-size 
fits  all'  approach  to  SBT  was  unlikely  to  be  effective.  This  is  because  individual 
differences  are  known  to  impact  on  training  outcomes  using  traditional  methods,  so 
this  is  also  likely  to  be  the  case  for  SBT.  Participants  agreed  that  training  is  likely  to  be 
more  effective  when  tailored  to  individual  learning  needs  and  capabilities.  For 
example,  computer-based  self-paced  instruction  (a  form  of  SBT)  may  be  more  effective 
for  individuals  who  are  highly  self-motivated. 

Overall,  participants  believed  that  further  research  is  required  to  address  questions 
regarding  the  cost-effectiveness  of  SBT.  Furthermore,  this  research  may  take  considerable 
time,  resources  and  client  engagement,  particularly  if  the  long  term  effectiveness  of  SBT  is 
to  be  reliably  measured. 

How  would  you  engage  with  industry  in  order  to  achieve  best  return  on  investment  for 
simulation-based  training? 

The  majority  of  participants  had  little  previous  experience  engaging  with  the  simulation 
industry.  In  addition,  there  were  no  representatives  from  industry  at  the  workshop  to 
inform  the  discussions  on  this  topic.  Consequently,  there  was  limited  discussion  regarding 
this  question,  although  the  following  points  were  noted. 
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•  It  was  generally  believed  that  COTS  solutions  lacked  sufficient  flexibility  to  meet 
training  requirements.  It  was  noted  that  there  is  often  no  access  to  software  source 
codes  and  limited  opportunities  to  tailor  scenario  design  to  meet  training 
requirements5.  Participants  believed  these  issues  need  to  be  addressed  as  part  of  the 
acquisition  process;  otherwise  this  can  result  in  significant  additional  costs  to  the  client 
such  as  payments  to  the  manufacturer  for  software  development. 

•  It  was  agreed  that  there  is  likely  to  be  value  in  leveraging  and  using  technology  from 
the  gaming  industry.  However,  it  was  noted  that  game-based  engine  models  may  not 
be  sufficiently  mature  to  meet  military  training  requirements  and  may  also  need 
additional  development6. 

•  Participants  believed  that  manufacturers  of  simulation  technologies  were  more  focused 
on  incorporating  extraneous  features  ("bells  and  whistles")  that  sell  the  'realism'  of 
simulators,  even  though  they  may  have  limited  or  no  training  benefit.  It  was 
acknowledged  that  industry  is  driven  largely  by  product  sales,  which  often  relies  on 
the  ability  to  sell  increased  realism  to  the  military  customer.  Given  this  situation  is 
unlikely  to  change,  it  was  recommended  that  defence  clients  need  to  be  more  rigorous 
in  the  acquisition  process  (as  noted  previously)  to  ensure  they  are  getting  value-for- 
money.  It  was  noted  that  the  research  community  can  assist  defence  clients  in  this 
process  by  helping  determine  the  features  in  the  simulation  that  are  necessary  for 
achieving  training  benefits.  This  includes  ensuring  that  these  devices  provide 
appropriate  feedback  to  trainees.  While  many  simulators  developed  by  industry 
include  tools  for  performance  analysis  and  feedback,  if  the  feedback  provided  by  the 
simulator  about  trainee  performance  is  not  relevant,  timely  or  accurate,  the  training 
value  of  such  tools  will  be  limited.  In  the  worst  case,  such  information  may  even 
contribute  to  negative  training  outcomes. 

Where  should  future  research  in  simulation-based  training  focus?  Where  are  the  high 
pay-off  areas,  and  why? 

Overall,  participants  believed  that  future  research  needed  to  continue  to  build  up  the  body 
of  evidence  relating  to  the  effectiveness  of  SBT.  However,  there  were  also  several  specific 
areas  identified  for  future  research.  These  included; 

•  Modelling  of  human  behaviour  in  simulation  systems.  Participants  believed  that 
training  non-kinetic/ social  skills  (e.g.,  cultural  awareness,  interpersonal  skills)  was  an 
important  area  given  the  nature  of  current  military  operations.  Current  methods  for 
training  these  skills  are  often  resource  intensive  (e.g.,  use  of  human  role  players).  It 
was  agreed  that  technology-based  solutions  could  offer  more  convenience  and  require 
fewer  resources  than  traditional  methods  for  this  type  of  training.  However, 


5  The  simulators  developed  and  used  by  Monash  University  Accident  Research  Centre  (MU ARC) 
for  civilian  road  safety  research  were  mentioned  as  good  examples  of  overcoming  limitations  with 
current  COTS  simulators. 

6  For  example,  the  COTS  game  Operation  Flashpoint  was  modified  into  VBS2  in  order  to  meet  the 
requirements  of  the  military  customer. 
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participants  believed  more  research  was  still  needed  to  ensure  the  representation  of 
human  behaviour  in  these  simulation  technologies  was  sufficiently  accurate  to  support 
effective  training  of  these  skills. 

•  Validation  of  simulators  and  human  performance.  Participants  highlighted  that  SBT 
research  that  examines  human  performance  appears  to  be  based  on  the  assumption 
that  the  level  of  performance  (and  operator  states)  observed  in  the  simulated 
environment  is  equivalent  to  (or  representative  of)  the  performance  that  occurs  in  the 
real-world.  This  assumption  may  not  always  be  supported,  and  participants  suggested 
that  validation  of  simulators  is  an  area  for  future  research  to  ensure  that  performance 
measurement  in  SBT  is  a  reliable  indicator  of  real-world  performance.  Such  validation 
studies  would  allow  researchers  to  conduct  future  SBT  research  with  greater 
confidence  that  the  outcomes  are  generalisable  to  the  real-world.  This  research  could 
include  assessing  the  limits  of  trainee/ operator  performance  using  SBT  as  was 
highlighted  earlier. 

•  Evaluating  the  utility  of  simulation  for  training  cognitive  skills.  Participants 
highlighted  that  due  to  the  complex  environments  in  which  military  personnel  operate, 
there  is  a  need  to  invest  in  training  critical  thinking  and  decision-making  skills,  as  well 
as  the  ability  to  cope  with  ambiguity.  Participants  acknowledged  that  training  such 
skills  probably  works  best  when  the  training  environment  is  supportive  of  making 
mistakes  and  provides  opportunities  for  reflection  and  corrective  feedback.  It  was 
believed  that  simulation  could  be  used  to  help  facilitate  the  training  of  these  specific 
skills  and  the  potential  for  microworld  simulations  was  again  highlighted. 

•  Exploring  the  use  of  simulation  for  complex  and  high-risk  missions.  Participants  noted 
that  SBT  can  be  used  to  expose  trainees  to  more  complex  and  high-risk  scenarios  than 
might  be  possible  in  live  training  due  to  safety  and  financial  constraints.  Consequently, 
participants  believed  that  SBT  could  potentially  be  used  to  explore  limits  of  human 
performance  in  a  safe  environment,  and  increase  trainee  confidence  and  competency 
levels  for  certain  skills  prior  to  conducting  them  on  operations.  It  was  noted,  however, 
that  the  ability  to  take  more  risks  in  the  simulator  may  lead  to  similar  risks  being  taken 
in  the  real  world  with  adverse  effects  (i.e.,  negative  training).  Notwithstanding  this  last 
point,  participants  felt  that  future  research  should  explore  this  area  further. 

3.3  Constraints 

The  number  of  workshop  attendees  was  limited  to  30  participants  on  a  'first  come,  first 
served'  basis  due  to  the  venue  capacity.  Also,  major  budget  restrictions  within  Defence  at 
the  time  of  the  workshop  limited  the  ability  of  personnel  to  travel  interstate,  and  VTC 
facilities  were  not  available  at  the  venue.  The  authors  acknowledge  that  these  factors  may 
have  excluded  the  attendance  of  some  stakeholders  and  hence  their  viewpoints  may  not  be 
captured  in  this  document.  In  particular,  there  were  no  ADF  representatives  at  the 
workshop  with  a  specific  interest  in  SBT.  Notwithstanding  this,  there  was  a  good 
representation  of  personnel  with  significant  knowledge  and  experience  in  the  field  of  SBT 
research,  so  we  are  confident  the  views  captured  reflect  the  current  state  of  knowledge  in 
the  field.  Indeed,  the  key  themes  that  emerged  from  the  workshop  were  consistent  with 


10 


UNCLASSIFIED 


UNCLASSIFIED 

DSTO-GD-0754 

the  authors'  knowledge  of  SBT,  as  well  as  recent  literature  in  this  area  (see  the  Reference 
list  at  the  end  of  the  report  for  examples). 

Due  to  time  constraints,  it  was  not  possible  to  distribute  the  workshop  questions  to 
participants  in  advance;  this  may  have  limited  the  depth  of  discussion  during  the 
workshop,  together  with  the  limited  time  that  was  available.  If  similar  workshops  are  held 
in  future,  it  is  recommended  that  at  least  a  full  day  is  allocated  to  allow  sufficient  time  for 
presentations,  discussions,  and  networking.  Furthermore,  it  is  recommended  that 
workshop  organisers  distribute  any  background  material  to  attendees  well  in  advance  to 
ensure  that  participants  arrive  as  prepared  as  possible  and  to  maximise  the  time  available. 

3.4  Additional  Material 

The  reference  section  of  this  document  contains  a  list  of  papers  that  may  be  of  interest  to 
the  reader.  It  is  not  intended  to  be  an  exhaustive  list  of  references  but  rather  provide  the 
reader  with  additional  resources  relating  to  some  of  the  themes  and  presentation  topics 
covered  in  the  workshop.  These  papers  cover  the  following  topics:  reviews  of  simulation- 
based  training  and  education  [1-5];  training  of  cultural  awareness,  adaptability  and 
leadership  skills  using  simulation  [6-7];  performance  measurement,  simulator  fidelity, 
transfer  of  training,  and  training  evaluation  [8-12]. 

4.  Conclusion 


The  workshop  provided  a  valuable  opportunity  to  bring  together  stakeholders  working  in 
the  area  of  SBT  and  to  capture  their  viewpoints  and  share  lessons.  The  four  main  questions 
provided  an  organising  structure  for  the  workshop  and  several  key  lessons  emerged  from 
the  presentations  and  discussions.  These  lessons  are  relevant  to  both  researchers  and 
defence  stakeholders. 

•  More  empirical  research  is  needed  in  order  to  improve  the  quality  of  advice  that  can  be 
provided  to  clients  regarding  the  cost-effectiveness  of  SBT;  in  particular 

o  better  understanding  the  relationship  between  simulator  fidelity  and  the 
effectiveness  of  SBT; 

o  determining  which  skills  are  more  effectively  trained  using  SBT; 
o  measuring  differences  in  skill  acquisition  and  retention  rates  for  SBT  and  other 
training  methods;  and 

o  developing  more  reliable  measures  of  effectiveness  for  SBT. 

•  Simulation  offers  potential  advantages  over  conventional  training  in  a  number  of 
areas;  high-pay  off  areas  for  future  research  include: 

o  training  beyond  competency  and  exploring  the  limits  of  operator  performance; 
o  training  for  complex  and  high-risk  missions;  and 
o  training  critical  thinking,  decision  making  and  social  skills. 

•  Basic  principles  of  training  and  education  still  need  to  be  applied  to  achieve  effective 
outcomes,  including: 

o  use  of  guided  instruction;  and 
o  consideration  of  individual  differences. 
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•  The  acquisition  of  SBT  capability  should  follow  a  more  rigorous  process  to  ensure 
'value-for-money',  including: 

o  clearly  articulating  training  requirements; 

o  using  training  requirements  to  identify  potential  training  solutions  and  inform 
simulation  specifications;  and 

o  conducting  a  cost-benefit  analysis  to  compare  different  training  options, 
including  consideration  of  alternatives  to  SBT  such  as  simple  training  aids. 

Overall,  more  empirical  studies  are  needed  before  the  questions  covered  in  the  workshop 
can  be  fully  addressed,  and  the  research  community  is  able  to  provide  more  informed 
advice  to  defence  clients  about  the  use  of  SBT.  Despite  the  limited  time  available, 
participants  believed  that  the  workshop  was  valuable  in  raising  their  awareness  of 
research  being  conducted  into  SBT  and  discussing  some  of  the  key  issues  in  this  area.  The 
above  lessons  will  provide  a  useful  starting  point  for  further  discussion  regarding  SBT 
between  researchers,  defence  clients  and  industry. 
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Appendix  A:  Workshop  Attendees 

The  following  table  lists  the  workshop  attendees  and  their  affiliation. 
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First  Name 

Organisation 
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Appendix  B:  Summary  of  DSTO  studies 

Utility  of  VBS2  Simulation  for  Individual  and  Collective  Training 

Dr  Susannah  Whitney  (DSTO  LOD)  gave  a  presentation  on  two  studies  looking  at  the 
effectiveness  of  the  first-person  simulation  VBS2  for  training  military  skills.  The  first  study 
looked  at  the  use  of  VBS2  for  training  section  attack  to  novice  infantry  trainees,  which  is  an 
example  of  collective  training  in  a  collective  context  [13].  The  study  compared  the  pre-  and 
post-training  performance  of  two  teams  at  section  attack,  where  one  team  had  conducted 
training  using  VBS2,  and  the  other  team  had  conducted  training  in  the  field  (i.e. 
conventional  training).  The  study  outcomes  showed  that  while  the  performance  of  the 
field-based  training  group  increased  significantly  after  training,  there  was  no  change  in 
performance  for  the  group  trained  using  VBS2.  This  outcome  shows  that  the  simulation 
has  little  utility  for  training  section  attack  procedures  for  this  population  of  trainees.  The 
main  reasons  for  this  outcome  were  limitations  in  the  simulator  fidelity,  relative  to  the 
requirements  for  the  task  being  conducted.  Specifically,  there  was  a  very  large  difference 
in  the  physical  skills  required  to  conduct  section  attack  in  the  field,  compared  with  how 
these  skills  were  executed  in  VBS2.  In  addition,  the  narrow  field-of-view  did  not  allow  the 
degree  of  team  awareness  required  for  this  collective  training  activity.  The  overall 
implication  of  these  outcomes  is  that  (1)  VBS2  may  have  limited  utility  for  other  collective 
infantry  tasks  and  (2)  the  nominally  cheaper  method  of  training  was  not  effective. 

The  second  study  looked  at  the  use  of  VBS2  for  Explosive  Hazard  Awareness  Protection 
Training,  where  the  aim  is  to  train  vehicle  convoy  commanders  to  respond  to  explosive 
hazards  [14].  In  contrast  to  the  section  attack  task,  this  is  an  example  of  individual  training 
in  a  collective  context.  The  study  compared  the  training  outcomes  for  traditional  training 
and  VBS2  and  provided  some  evidence  to  suggest  that  VBS2  was  a  useful  supplement  to 
traditional  training  methods.  Some  of  the  limitations  in  VBS2  identified  during  the  section 
attack  study  (e.g.,  limited  field-of-view)  were  also  identified  during  this  study,  but  did  not 
appear  to  be  as  critical.  The  two  studies  highlighted  that  the  utility  of  a  given  simulation  is 
dependent  on  the  task  being  trained.  In  these  cases  VBS2  was  found  to  be  ineffective  for 
collective  training  in  a  specific  collective  context  but  showed  promise  for  individual 
training  in  a  different  collective  context.  The  implication  here  is  that  it  is  critical  to  specify 
the  task  and  context  in  which  the  training  is  conducted  and  not  simply  describe  a 
simulation  as  being  effective  (or  not  effective)  for  training. 

Utility  of  Simulation  for  Pilot  Training 

Dr  Greg  McLean  (DSTO  AOD)  gave  a  presentation  on  the  use  of  SBT  in  an  ab  initio  pilot 
training  course  [15].  The  initial  findings  of  the  study  demonstrated  that  use  of  the 
simulation  resulted  in  a  reduction  in  live  flying  hours,  but  an  increase  in  total  training 
time.  Overall,  there  was  little  difference  in  cost  between  an  all  live  and  blended  training 
program,  although  students  may  have  felt  more  confident  with  initial  familiarisation  in  the 
simulator  rather  than  going  straight  to  the  real  aircraft.  This  was  an  interesting  finding, 
given  that  use  of  simulation  in  the  air  domain  is  often  held  up  as  an  example  of  the  cost- 
benefit  advantages  associated  with  SBT.  The  lesson  here  is  that  it  should  not  be  assumed 
that  use  of  simulation  automatically  achieves  cost-savings. 
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Use  of  Simulation  for  Command  and  Control  Team  Training 

Dr  Greg  McLean  (DSTO  AOD)  gave  a  second  presentation  on  the  use  of  synthetic 
environments  as  preparation  for  Command  and  Control  (C2)  teams  in  live  exercises  [16]. 
Exercise  Black  Skies  2012  (EBS12)  involved  Air  Battle  Management  (ABM)  and  Airborne 
Early  Warning  and  Control  (AEW&C)  Teams  conducting  synthetic  training  scenarios 
matched  to  the  missions  they  would  conduct  eight  weeks  later  at  a  live  exercise  (Pitch 
Black  12:  PB12).  During  EBS12,  the  performance  of  both  teams  improved  across  a  number 
of  mission  objective  and  teamwork  dimensions,  consistent  with  the  training  being 
effective.  Furthermore,  the  subsequent  performance  of  the  EBS12  AEW&C  team  during 
PB12  provided  evidence  that  the  benefits  of  EBS12  transferred  to  the  live  exercise.  It  was 
noted  that  in  order  to  achieve  this  outcome,  a  4:1  ratio  of  white  force  to  exercise 
participants  was  required;  that  is,  the  training  audience  was  only  20%  of  the  total  number 
of  participants.  The  implication  is  that  the  conduct  of  such  realistic  training  is  highly 
resource  intensive,  and  hence  may  not  be  achievable  on  a  frequent  basis  if  the  client 
incorporated  these  methods  into  regular  training.  Subsequent  research  will  examine 
whether  this  ratio  can  be  reduced  (e.g.,  through  use  of  automation,  simulated  white  force, 
non-expert  role  players)  without  compromising  the  training  benefit. 

Use  of  Simulation  in  Military  Education 

Mr  John  Hansen  gave  a  presentation  on  the  use  of  simulation  for  professional  military 
education,  in  particular  some  work  conducted  within  DSTO  C3ID  looking  at  the  use  of 
microworlds  for  exploring  C2  issues  [17],  The  presentation  discussed  the  characteristics  of 
a  good  microworld  simulation  and  how  this  medium  could  be  used  for  making 
individuals  aware  of  their  own  particular  thinking  styles  and  how  this  impacted  on  their 
decision  making.  It  was  noted  that  certain  thinking  styles  are  characterised  by  a  number  of 
cognitive  biases  and  are  unsuited  to  dealing  with  uncertainty  and  ambiguity;  conditions 
that  are  often  faced  by  decision  makers  in  the  military  domain  (as  well  as  other  domains). 
The  implication  is  that  microworld  simulations  could  be  used  to  help  military 
commanders  and  leaders  to  improve  their  decision  making  abilities.  The  presenter  also 
noted  that  some  of  the  issues  that  are  relevant  for  SBT  will  also  be  important  for  education; 
however  there  will  also  be  differences.  This  point  was  taken  up  in  a  subsequent 
presentation  by  Dr  Daniel  Goodburn  (also  from  C3ID),  who  discussed  some  of  the  key 
points  from  a  review  of  the  field  of  simulation  in  education  [4],  The  review  noted  that 
future  work  could  potentially  focus  on  designing  simulation  which  enables  a  more 
facilitated  approach  to  education  in  which  expert  practitioners  and  students  explore 
complex  problems  in  order  for  the  student  to  learn  how  to  think  (rather  than  what  to 
think).  This  idea  aligns  with  the  research  into  the  use  of  Microworlds  for  improving 
decision  making  abilities  outlined  previously.  At  present,  this  capability  appears  to  be 
underutilised.  The  implication  is  that  there  are  opportunities  being  missed  in  terms  of 
enhancing  educational  practices  through  the  use  of  simulation. 
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